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ABSTRACT 

This paper raises three broad issues,each with 
sub-areas. First, what are the consequences of measuring and 
evaluating the parts of higher education that can be satisfactorily 
measured? What are the consequences for the parts of education that 
can not be measured, and for the total enterprise? Second, what kind 
of assessment systems will be best in the long run in providing 
incentives for higher quality, better performance, and greater 
innovation amd flexibility? Should we attempt to incorporate the 
standards into the budget process, or should we emphasize program 
review and audit? Third, what are the appropriate roles in the 
process for the public, legislators, state higher education agencies, 
governing boards, and institutional administrators, faculty, and 
students? Can we devise a set of relationships that leave the central 
responsibility for the educational program with the educational 
institutions, but provides for appropriate consideration and input 
from the other concerned groups? What is appropriate concern and 
input? These questions have a new urgency today, because higher 
education is being influenced by internal and external pressures that 
are likely to produce incomplete and unsatisfactory answers. 
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Educators, legislators, and other government officials have a high level 
of interest in the topics of performance budgeting and performance audits in 
education. A recent priority rating of the importance of 21 current educational 
topics by ECS commissioners, ranked educational finance first, the effectiveness 
of basic education second, and educational accountability fourth in importance. | 
Since ECS commissioners include a broad representation of legislators and governors, 
as well as educators at all levels, this high ranking of topics that are central 
to performance budgeting is a useful barometer of the wider climate of opinion. 
This paper is in two sections. The first reviews some of the reasons for 
the increased interest in performance budgeting and performance auditing, and 
the second reviews some of the issues for discussion and resolution in anticipa- 
tion of further efforts in relating both budgeting and auditing to the outcomes 


of education. 
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Since this field is full of ill-defined terms and code words, it may 
be well to begin with a couple of definitions, so that you know what I mean 
by these terms. A performance or outcome budget is one that includes some part 
of the funding on the basis of the quality and outcomes of the activity, rather 
than being based solely on the quantity of activity, or on historical levels of 
support. A program budget only becomes a performance budget when its funding 
is related to the program quality or outcomes -- just classifying costs by 
programs doesn't make a performance budget. A performance audit is an assessment 
of the effectiveness of an educational program or institution. It is more than 
a management audit, which deals with questions of efficiency. A performance 


audit is also concerned with outcomes and the effectiveness of goal achievement. 


I 


Who's Interested in Performance Budgeting 


and Performance Auditing, and Why? 
It will be useful to divide the question above into two parts. The first 


will deal with the external pressures for better outcome measures, and the 
second part will try to identify those influences within education that are 
interested in performance measures. 

A major external factor is the new environment within which postsecondary 
education, as well as other social institutions, operates today. There are 
several characteristics of the new environment. a 

Public opinion polls have charted the general decline of public confidence 
in the effectiveness of all of our social institutions; health care, legislatures, 
the public schools, and higher education. The public is less sure of the 


ability of the various institutions to carry out their responsibilities, whether 
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it be to make laws, educate children, control crime, or cure mental illness. 

While education still receives relatively high marks as compared with other 

social institutions, the public concern and skepticism affects all-institutions. 
It also makes the legislature more skeptical and critical; part of their job today 
is to demonstrate to the public they too are concerned, and that they are 

making laws and taking steps to see that the public institutions shape up, don't 
waste money, and provide better public services. 

In this country we have had a theory of separation of neuen between the 
three branches of government and between government in general and certain key 
institutions such as the press and education. This is based on the widely held 
belief that the content of education and communication must not be controlled by 
the government in a democracy. 


The legislature has the task of getting higher education to do a more 


effective job, and to raise the level of public confidence in the educational 
process, in a situation where their historic relationship with education has 
been indirect and mediated through lay boards with considerable autonomy in the 
public sector, and with almost complete autonomy in the independent sector. 

Thus one might expect that the legislature's approach to getting postsecondary 
education to do a better job would be different from the approach that they use 
with the Department of Corrections or the Welfare Department. However, there 
seems to be an increasing jnotination to treat public and higher education like 
other parts of state government. This tendency is related to another public 
attitude, that the autonomy enjoyed by doctors, professors, and other professionals 
has bean used more to the benefit of the professional group than of the public 
generally. Combined with the egalitarian trend the legislative and governmental 
‘skepticism about self-regulated professional groups leads to a legislative and 


executive position that what is good enough for the highway department, insofar 
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as being accountable for the effectiveness of their work, is also good enough 
for the colleges and universities. 

Another reason for an interest in accountability and outcome measures is 
the increasing size of the higher education budget. The percent of the state 
budget devoted to higher education approximately doubled between 1960 and 1973.2 
In addition, the legislature has more discretion about appropriations to higher 
education than to many other governmental functions, which may be determined 
statutorily or be heavily influenced by federal regulations and matching fund 
requirements. Thus, higher education funding is larger and more visible today 
than it was a decade ago, and it is subject to more executive and legislative 
discretion; both in expanding it when it is popular and when state revenues are 
growine, and in contracting it when popularity decreases or state revenues are 
declining. 

Finally, a key environmental factor in the last two years has been the 
continuation of recession and inflation which has made a reexamination of priorities 
in all areas of the state budget necessary, and has made it necessary for executive 
budget offices and legislative budget committees to require new evidence of 
efficiency and effectiveness. "Trimming the fat out" has become more than rhetoric 
in most states in the last couple of years. 

Goal-Oriented Budget Reform. 

Another external influence on educational outcome measurement has been the 
budget reform movement. The weaknesses and li:sitations of historical-incremental 
budgeting have been recognized for a long time, and beginning with the early 


sixties a number of new approaches to budgeting were proposed. While these had 


261enny and Kidder, State Tax Support and Higher Education 1963-73 
ECS, April 1974. 
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a variety of names, such as PPBS, Zero-Based Budgets, Management by Objectives, 
Program budgeting, etc., most of them were based on relating budgets and 
expenditures to specific objectives or goals. The theory was that by requiring 
clear identification of goals it should be possible to relate decisionmaking 

and expenditures to them, and made the budget a much more effective planning and 
management tool. 

Goal-oriented budget systems assume that goals can be identified clearly, 
that expenditures can be related to them, and that progress toward the goals can 
be assessed either objectively, or by some agreed upon subjective procedure. It 
has been difficult to fit these assumptions to some of our social institutions, 
including education. 

A number of states have adopted program-budgeting, PPBS, or some other 
variant on them, but in general, these changes have had a very limited impact 
on higher education. While institutions in states that have adopted program 
budgeting have made changes in the budget format, and have identified programs, 


and allocated costs to them, the impact on the planning, decisionmaking, and 


resource allocation methods of colleges and universities has generally been limited. 


There has been more change in the form of the budget than in the substance or the 
process by which it is developed and evaluated. 

Why, if all of the environmental changes described above have been taking 
place, hasn't budget reform had more impact on higher education? The answer to 
this question will be explored in more detail in the second half of this paper 
where the issues for the future are examined. However, higher education institu- 
tions have multiple and general, rather than specific goals, and universities are 


more confederations of individual professionals, each setting their own priorities, 


than hierarchical organizations with simple goals. The organization doesn't fit 
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the assumptions of the budget system very well, although both MBO and program 
budgeting systems accommodate to individual and sub-unit goals. Despite the 
limited impact of program budgeting in most states, a few states, most notably 
Hawaii, have emphasized program budgeting which includes outcome measures and 
assessment of progress toward objectives. 

Accountability 

Another external influence has been the growth of the accountability 
movement. As public higher education budgets have grown, and particularly in 
the recent turndown in the economy and in tax collections, there has been an 
increased gubernatorial end legislative concern with the efficiency of all state 
activities, including educational institutions. State auditing activities have 
been expanding from fiscal audits to include annanement audtbe. and from the 
management audit, it is only another step to the performance audit, which examines 
outcomes and the effectiveness of programs. The Federal General Accounting Office 
has been involved in performance audits for some time, and an increasing number 
of states have begun to carry out performance audits. Performance auditing has 
been concentrated in the larger states, with better developed audit staffs, but 
is spreading to a number of medium sized and smaller states. 

Auditing is not the major approach to increased efficiency. Several state 
legislatures have sought to impose more efficiency, by mandating heavier teaching 
loads, year-round calendars, better use of space, etc., as a part of the budget 
process. A more common approach, however, is for the legislative or executive 
budget process to simply cut the budget back across the board, leaving to the 
affected institutions the problem of where to make their operation more efficient. 
As far as I know, none of these external pressures for more efficiency has tried 


to use any effectiveness criteria or outcome measures as a way to reduce the 
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budget, or to examine in any systematic way what the impact of budget cuts was on 


the effectiveness of operations. 


Across the board budget cuts are politically expedient, both within the 


institution, as well as on the part of the executive and the legislature, if the 


assumption is that the fiscal set-backs are temporary. Frank Bowen's study of 
states where higher education budgets have been cut shows that when institutions 
think financial difficulties are likely to continue for a considerable number of 
years, they then begin to look for ways to make selective reductions, and to 
consider qualitative issues, as revealed by the New York and Wisconsin program 
review processes. Fiscal retrenchment has generated a climate conducive to the 
devalopment of efficiency measures, and this same executive and legislative 
concern creates an interest in effectiveness measures. 

At the elementary and secondary level, the Cooperative Accountability Project 
has kept track of legislation relating to accountability®, and in their most 
recent report identified 31 states that had enacted some sort of accountability 
legislation. Their definition of accountability legislation is quite broad, and 
includes such things as establishing a uniform accounting system, doing comprehen- 
sive planning, and setting up a management information system. There are twenty 
three states that have enacted legislation to create, at the elementary-secondary 
level one or more of the following: A system of program evaluation, competency 
based teacher certification, and/or formal assessment of student achievement. 

It thus appears that about half the legislatures have mandated development of 
some form of outcome measures in elementary-secondary education. Just how these 


efforts will be related to the budget process is not clear, but it is evident 


that there is a lot of interest in performance outcomes as a part of accountability. 


3cooperative Accountability Project, Legislation by the States: Accountability 
and Assessment in Education, State Education Accountability Repository, 
Wisconsin Department of Public Instruction, 1976 
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These external influences on the development of performance budgets and 
performance audits haven't resulted in many tangible outcomes yet in higher 
education, but they seem quite likely to in the future. They are getting reinforce- 


ment from some sources within postsecondary education, which will be reviewed 


in the next section. 


Innovative Programs and Performance Criteria 


There are several influences within higher education that are concerned with 
performance criteria. One of the most important of these is the administration of 
nontraditional and innovative programs. These programs can't be easily equated 
to traditional credit-hour measures. Nontraditional programs need some performance 
standard to demonstrate the equivalence of their programs with regular on-campus 
programs, and this is especially true of those programs conducted entirely off-campus 
and/or without contract with regular faculty. Thus a variety of innovative 
programs need performance criteria to validate the quality of the educational 
experiences they provide. Often outcome measures are the only evidence they have -- 


since their programs don't meet the usual criteria of time or faculty contact. 


The problem is that regular programs don't usually have comparable outcome measures, ~ 


so the innovators also become a major lobby for performance measures that will 
be used across the board. Innovators are often concerned that their programs 
won't be funded adequately through the usual credit-hour oriented formulas used 
by many states, and thus the introduction of performance criteria seems to hold 


out the possibility of more adequate funding.” 


Research -- An Example of Performance Funding 


In one area, the funding of research in universities, performance criteria 


have been widely used in deciding on funding for projects and programs. 


Ls Richard Meeth. Government Funding Policies and Nontraditional Programs 
Washington, D. C., Institute for Educational Leadership. June, 1975 


10 


ae 


Fund requests are usually reviewed in detail by peers in the same fieTd, and are 
awarded in competition with other research projects. The results are also 
reviewed, and reports from research are generally refereed before being published. 
Ineffective performance is likely to lead to no new grants, and if bad enough, 
to termination of existing grants. A majority of research funding in universities 
is provided through competitive awards, which attempt to assess by a variety of 
objective and judgmental procedures, the potential effectiveness of the proposed 
research. Research is at least as complex a process as teaching and may be even 
harder to assess for funding purposes. But in this broad area of university 
activity we have a performance-oriented funding system that assesses outputs and 
judges the quality of very complex tasks, even though research has objectives 
that are often quite vague. R2search funding is of course not usually a part of 
the "regular" budget, although the regularity of funding of research has been 
pretty high, when considered for the total national support of research in ; —< 
universities. 

Stable funding of the university's general fund budget makes the competitive 
performance -oriented funding of research easier, by assuring the researcher of 
a base of support, through his teaching, but in theory a similar system could be 
extended to the teaching as well as the research budget. The sociologist 
Theodore Caplow proposed to an institutional research conference about 20 years 
ago that teaching be budgeted on a performance contract basis similar to the 
research grant, but this idea is notable primarily because it has had almost no 
impact on educational budgeting up to this point. In view of the obvious parallels 
between research and teaching, the key question is why haven't people tried the 


same funding approach in teaching that they have in research? 
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Program Evaluation and Outcome Measures Within the Institution 
Another source of interest in assessment and outcome measures within higher 


education are the administrators of institutions who are anticipating a long-term 
requirement for strict budgetary controls, and reallocation of resources from one 
program area to another. The Wisconsin System has initiated a process of planning 
and program review which is designed to assess the effectiveness of programs, as 

a basis for budgeting decisions to cut back, maintain, or expand the preara” 

A number of private institutions have initiated internal budget processes which 
require schools within the university to do long range budgeting in relation to 
specific outcome objectives. Where the prospects for fiscal stringency are long 
term, the internal response in a number of institutions is likely to be an 
assessment process that utilizes outcome measures, and which evaluates programs 


against performance criteria. 


~~ Student Pressures for Performance Evaluation 

In the past, the large majority of students have expressed general satis- 
faction with their education, and by their continued enrollment have provided 
a kind of "bottom line" assessment of education. The rise of programs of student 
assistance to increase student access and choice of educational options has also 
raised questions about the adequacy of information to students on which they can 
make informed choices. Inevitably, this gets into information about the outcomes 
and effectiveness of the programs that the students are choosing. In the future, 
there are likely to be greater requirements for information about outcomes as 
part of the consumer protection, student information for choice emphasis in 


postsecondary education. 


Spon K. Smith "Academic Program Audit and Review as a Means of Resource 
Reallocation," Education Commission of the States, IEP Seminar, December 16, 1975. 
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Accreditation and Outcome Budgeting 


The regional accreditation process turned away from the use of objective 
criteria (such as minimum faculty salaries, minimum expenditures per student, etc.) 
as a major basis for accreditation fifteen or twenty years ago, and substituted 
a self-evaluation process, in which each institution was to be judged against 
its own goals and objectives. Judgments were largely subjective evaluations of the 
processes being followed, rather than the outcomes being achieved. The process 
was largely internal and confidential, with only the final result being made 
public. Accreditation in general does not seem to have been a very important 
influence leading to the development of outcome measures or performance audits. 

It has concentrated on the processes being followed, rather than on the results 
being achieved. Although it is in the nature of an audit, the internal and 
confidential nature of the process is such that it doesn't satisfy external 
pressures for public performance audits, nor have accreditation results been a 
significant part of budget processes, except in the case of medical education, 
where they are essential to the release of federal funds. 

The preceding review has identified a number of internal to education 


and external influences that are likely to cause more emphasis on both performance 


issues created by these developments, which should be of concern to educators, 


audits and on the use of outcome measures in education. What are some of the 
state higher education executives, budget officers, gubernatorial staff, and 
; 


legislators? The next section focuses on these questions. 


II 


Issues in the Development of Performance Audits 
and/or Performance Budgets 


A first question to be considered is whether we can do an effective job of 


assessing the important objectives in education. Educators have long been skeptical 
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about the possibility of developing objective measures of the complex processes of 
education that will not distort it, or lead to overemphasis on the part that 

can be measured, to the detriment of the parts (possibly more important) that 
can't be measured. 

It is not only the question of what outcomes can be measured, but also 
the question of how we should approach the whole process of evaluation. Even if 
we have a series of test scores of the graduates of all the programs, the outcomes 
of education are more than the sum of those scores. Does our approach to assess- 
ment take account of this additional complexity? As Bob Pace observed: 

"If (we)...are to deal effectively with large problems such as the 

effectiveness of a total institution, of a class of institution, 

or of higher education in the United States -- then we need a 

concept of evaluation that is both more comprehensive and more 

flexible than the familiar experimental model. For this level of 

complexity and reality, the evaluator must ask different questions, 

proceed in a different style..."6 

Pace goes on to say that evaluation of complex organizations must be 
concerned not only with intended outcomes, but with the other consequences, that 
it must not attempt to reduce everything to a few simple measures. He also favors 
a separation of the evaluative function from the direct responsibilities for 
management or operations. 

This suggests that the introduction of outcome measures into the budget 
process will create ripples which extend beyond the area where the outcomes are 
being assessed. The case of funding research on the basis of competitive proposals 
in which outcomes are emphasized illustrates this very well. Research funds also 
provide employment for graduate students and have been a very powerful stimulant 


for the expansion of graduate programs, which in turn require expansion of the 


regular university instructional budget for salaries, equipment, space etc. 


6c. Robert Pace, Thoughts on Evaluation in Education. American College Testing ' 
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A complex chain of interdependent organizational effects is set in motion by change 
in one part of the organization. 

The issue is what the consequences will be for the total educational process 
if we only measure a few of the outcomes of education. Will a university that is 
successful in performance competition for research funds overemphasize graduate 
education and neglect undergraduate teaching? The usual answer to that question 
is "no" by university faculty and "yes" by outsiders. Will undergraduates be 
attracted to a graduate university by the prestige of the faculty and will they 
get a better education than at a strictly undergraduate institution. There are 
a lot of undergraduates who think so, but it's hard to assess those other 
- consequences by the usual approach to measuring performance. 

If we wait until we can measure all the important outcomes of education and 
all the consequences of change in one part of the organization for other parts of 
the organization, then we will never introduce performance budgeting. It is also 
clear that changing the criteria for determining part of the budget will have 
consequences for other parts of the budget. 

Should we move ahead with efforts to use such outcome measures as are 
available, recognizing that this will be a limited assessment of the total outcomes 
of the institution, and recognizing that a change in the funding pattern of one 
part of the institution can have unintended, but major, consequences for some 
other part of the organization? Or do we try to develop some more global form 
of institutional assessment? Or do we continue to argue that the costs are greater 
than benefits, and try to avoid outcome measures entirely? 

An answer to these questions depends on how you assess the advantages and 
disadvantages, and I hope you will try to answer those questions for yourself at 


this conference. 


2 Ths 
Performance Budgeting or Performance Auditing? 

Changing the budget process to include outcome measures may have different 
consequences for institutional management and for institutional autonomy than 
will the response to a performance post audit. The obvious analogy is between 
the impact of a fiscal post audit, and the introduction of additional line item 
controls in a budget process. Most people would agree that the introduction of 
more line item controls limits autonomy and managerial flexibility more than 
adherence to post audit standards, and the first response is likely to be that 
the post audit is preferable. 

The issue is more complex, however. Suppose performance criteria were 
introduced into the budget in the form of incentive grants -- much as they have 
been in research. Few would argue that the research contract and grant system 
unduly limits institutional autonomy, since the institution is free not to submit 
an application if they think the terms of the grant are bad or the kind of work 
is inappropriate. If an institution wants to be a major graduate university, 
however, it must apply for some grants, and once you have the grant habit, it is 
very difficult to give it up. 

Fiscal audit standards impose a number of restrictions on managerial 
fiexibility. They are not generally regarded as threats to autonomy, because we 
think the standards are reasonable and desirable, and because the institutions 
know what standards will be applied. This is not the case with performance audits. 
The standards or criteria which should be met are not a matter of consensus. 
What.is to be assessed, how it will be assessed, who will assess it, and what the 
budgetary consequences should be are all matters of disagreement. You can test 
that statement by raising the issues in a mixed audience of legislators, state 
fiscal officials, college administrators and faculty if you don't believe it. 


As a consequence, a performance audit may be viewed by the institution as improper 
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and unwarranted intrusion on the legal authority and responsibility of the 

institution to manage the educational program. At the same time it may be seen 

by the legislature as a necessary step in more effective public policy determination. : 
A case in point is the recent controversy between the University of Wisconsin 

and the Wisconsin legislature over a proposed audit by the latter agency. 

When you think about the potential impact of performance budgeting or 
performance auditing, both will affect institutional management, and will affect 
managerial autonomy. The way in which the processes are carried out, who carries 
them out, and the standards or criteria that are used will be far more important 
than whether we apply those performance standards on the front end, through the 
budget process, or after the fact, with an audit. Actually it is likely to be 
performance budgeting and performance auditing, for if the standards are applied 
in the budget process, the audit process must look to see if they have actually 


been applied properly. 


Who Should Determine the Performance Standards and How should the Process Operate? 
Self-evaluation through voluntary accreditation has been the traditional 


way that institutions have sought to validate the quality of their efforts. The 
emphasis in accreditation has been on elimination of the very weak -or fradulent 
institutions, and the encouragement of other institutions to improve themselves 
by establishing goals, and conducting self studies to see how they measure up to 
their own objectives. Most educators favor continuation of this self-regulating 
system as being most consistent with the maintenance of institutional control of | 
the educational process. | 

The process has been criticized in recent years because it doesn't. provide 
for input to goal setting and standards from the public, is ineffective in 


eliminating fradulent institutions, and doesn't provide enough incentive for 


institutions in the average and above average quality range to improve themselves. 
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The performance audit function raises the clear issue as to who should 
do it? Can an internal audit meet the needs of the public and of legislators who 
are concerned about the way that public funds are being used? Can an educational 
agency external to the institutions such as a state coordinating agency, or a 
state or federal licensing and approval agency perform the function? Or can the 
regular state audit agency, with appropriate staff supplementation, perform the 
function? What are the consequences if the responsibility is located outside the 
audit agency? 

If we want to talk about performance budgeting, then the same issue arises? 
Who sets the performance standards and criteria that will be used in the budget 
process? The peer review system in research grants and contracts has weaknesses 
too, which have been discussed at length, but it is essentially an external group 
of educators who make evaluations and set standards. But even though the standards 
for research review and evaluation are developed by educators, they are often 
applied within fairly narrow legal and administrative regulation boundaries, which 
greatly delimit the area in which educational evaluations can be made. 

The issues of who should be involved in the development of the standardé. 
for both performance budgets and performance audits is an important one which I 


hope you will discuss. 


Summar. 

This paper raises three broad issues, each with sub-areas. 

First, what are the consequences of measuring and evaluating the parts of 
higher education that can be satisfactorily measured? What are the consequences 
for the parts of education that can't be measured, and for the total enterprise? 

Second, what kind of assessment systems will be best in the long run in 


providing incentives for higher quality, better performance, and greater innovation 
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and flexibility? Should we attempt to incorporate the standards into the 
budget process, or should we emphasize program review and audit? 

Third, what are the appropriate roles in the process for the public, 
legislators, state higher education agencies, governing boards, and institutional 
administrators, faculty, and students? Can we devise a set of relationships 
that leave the central responsibility for the educational program with the 
educational institutions, but provides for appropriate consideration and input from 
the other concerned groups? What is appropriate concern and input? 

These questions aren't new. 1n one form or another they have been discussed 
for a long time. But they have a new urgency today, because environmental 
influences are pressing on higher education which are likely to produce incomplete 
and unsatisfactory answers. Assessments, and budget procedures are likely to 
emphasize efficiency rather than effectiveness; and cheapness rather than return 
per dollar invested, and concern for quality for the youth of America. I hope 
that this conference can help to assure a broader consideration of the important 
policy issues that are involved in stimulating innovation and desirable outcomes 


through the budget process. 


JKF:cja 


